Attention Correctness in Neural Image Captioning

Attention Map Visualization We visualize the attention maps of both the implicit attention model and our supervised attention model on the Flickr30k test set. As mentioned in the paper, 909 noun phrases are aligned for the implicit model and 901 for the supervised model. 635 of these alignments are common for both, and 595 of them have corresponding bounding boxes. Here we present a subset due to space. For every figure, the original image is on the left, the implicit attention result is in the middle, and the supervised attention result is on the right. The red box marks the ground truth attention region, as annotated in the Flickr30k Entities dataset. The attention correctness score for this phrase is in the parenthesis.